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ABSTRACT ^ 

Onder contract with the Ofegon State- Department of 
Education, Ed^jcational Testing -Service (ETS) assisted in the 
preparation of a fourth gra^e reading assessment test based ;on, or * 
modified from. Instructional Objectives Exchange materials* ETS 
produced test books, answer sheets, and dll required ancillary 
materials necessary for test distribution and administration to a 
sample of 104 schools, previously selected on the basis of geographic 
region, district wealth, and district size. Rights, Wrongs, and 
-"Don't .Know" pupil scores were reported to schools and districts for 
25 -specific reading objectives and 4 grouped objectives (word attack, 
vocabulary,- comprehension, and application) • Summary statistics for 
schools were also prepared and distributed, A special vocabulary 
self-jreport section/score was included in both reports. At the state 
level, a number of item analyses were performed, and a series of 
weight frequency distributions prepared, A total of 4,127 pupils at 
102 schools actually took the test with 299 pupils reported a^S 



absent, (Author) 
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The Sample 

At the time of the award of the contract for the Oregon Statewide. 
Assessment to Educational Testing Service, the saiq>le, consisting of 104 
independently drawn and randomly selected Oregon schdols, had already 
been identified under a separate* contt&ct between the Oregon Stjate Depart-- 
ment of Education and the Research Itlangle Institute (RTI)^« Ihd Oregon 
State Department of Education provided ETS with a list of school names 
and addresses, together with initial estimates of fourth grade enrollment, 

an Oregon-assigned identification number (^DE number) , and a special RTI 

. ^ ' ' / ' 

ideatifitation number specifying sanple stratification variables (geo^raph|c 

region, district wealth, district size)« The Oregon State Department of 

Education also provided the names and addresses of school principals, the 

Intermediate Education District (lED/County) Coordinators, and the district. 

superintendents who were to receive score reports. All direct contact^ 

with schools, districts^ and ISD/County^unl'ts regarding participation, 

score reporting, and similar matters were handled directly by the Oregon 

State Department Qf Education. 

Initial enrollment estimates for schools were later modified 'to reflect 

actual enrollment, and the quantities of materials adjusted to reflect 

these data. ' • 



•"•Prepared for presentation at the meetings of the National Council on 
Measurement in Education, Washington, D.Ci, April 2, 1975. 

^Bayless; David L. and Weber, Jllene: Sample Design oi Oregon Assessment 
of Fourth Grade Reading Performance in the Sptlng of 1973, Final Report 
22Cr-951, Research Triangle Institute Center for Educational Research and 
Evaluation, 19 pp. 
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The initial Oregon sample consisted of 104 schools » containing an 
estimated 4,544 fourth-grade pupils, located in 29 different Oregon lED/ 
County units. Revised figures, based on actiffa enrollment, increased the 
sample size to 4,584 pupils. . • . 
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The Test 



The initial pool of test items (questions) which constituted the 
^source of the actual Instrument administered to Oregon fourth-graders 
* was originally obtained by the Oregon State Department of Educdtlpn from, 
the Instructional Objectives Exchange* tlOX). Test items were initially 
selected and revised by the staff of the Oregon State Department of 
Education in order to provide measurement of twenty -five (25) specific 
reading objectives and four grouped objectives or "skills" areas (Word 
Attack, Vocabulary, Comprehension, and Application). 

The test items were systematically reviewed by ETS staff to confirm 
their relevance to the stated objectives, to minimize bias and possible 
offenslveness to mi^iorltles, and to judge their technical quality in terms 
of content and face validity. ETS staff suggested item revisions and mod- 
Iflcatlons and met with the Oregon State Department of Education staff and 
an outside reaching consultant to finalize the content and format of the 
test. 

In addition to the ninety-six (96) "standard" test items selected for 
inclusion in the test, a special vacabulary self -report section was also 
deve3|oped and Included. This section consisted of a list of vocabulary 
±p^3y drawn from the test and test directions, for whlcK the examinee was 
[to indicate whether the word was understood (i.e., "clear^or "not clear"). 
Following approval of the test, ETS designed a machlne-s cor able answer 
sheet for use on its NCS .equipmen^t; the answer sheet contained both the 



' actual questions and the 'answers (responses) for the first forty test 
* • - ■ 

items,' and provided response spaces for the last fifty-six items (con- 

- 4 

tained in a separately printed test book.) The answer sheet also con- 
tained the check7list of thirty-eight vocabulai^ items. 

Following test production and printing, *but prior to actual test 
administration, three i-tems were identified as faulty (items 13, 54, and 
55) and were deleted from all subsequei^t scoring, processing, reporting ^ 
and analysis. 

Pret^aration of Materials and Shipment 

As specified in its contract, ETS prepared and submitted a Basic 
Procedural Design Manual containing drafts of the form letters, manuals^ 
test books, answer sheets, and ancillary materials to be used in the 
administration of the program. Following review, revision, and final 
approval, appropriate quantities of each item were printed for packing 
and distribution. With the exception of one large shipment to Multnomah 
County, all packages were sent to lEn/County Coordinators via United. 
Parcel Service (UPS). Return shipments were automatically picked up by 
UPS on specified dates (April 30 for all lEDs testing one or two schools 
and May 3 for all lEDs testing three or more schools). UPS service ' 
attempts to pick up return shipment(s) on three successive days, beginning^ 
with the scheduled datfe^ and was to notify ETS if return shipments were 
unavailable. All packages were returned as scheduled and without additional 
follow-up. 

^ 

Training Sessions (Workshops) ' ' ' , , 

During the week of April 9-12, 1974, the staff of the Oregon State 
Department of Education, assisted J^y the ETS project director, conducted a 
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series of one-day training session/workshops In Wilsonville, Sutherlin, • 
and Pendleton. These meetings weW attended by lED/County, district' and 
school personnel, and provided general background to participants on the 
Oregon Statewide Assessment /rogram. The sessions also included a detailed 
review and discussion of the admloTstrative procedures to be utilized in 
the program. Saoq)le materials were distributed, reviewed, and discussed 
by the participants. Copies of the materials, specially prepared for the 
workshops, were made "available to the participants. 
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Test Administration • 

The test was to be administered to fourth-grade pupils in the selected 
schools on April 24, 1974. ^Pupils exempted from taking the test included 
those officially classified as blind, deaf, pliysically handicapped (unable 
to manipulate pencil and paper), trainable mentally retarded, or emotionally 
disturbed (fo the extent that testing them would have tended to disturb 
other pupils being tested)! Schools unable t« test on the specified date 
were asked to administer the Instrumeut during the week of April 22nd. 
In addition to admliiistering the test to pupils present on the test date. 



test administrators (teachers) were asked to (cwpplete .an answer sheet for 
those pupils absent on the date of testing, and to provide an indication 
that the pupil had been absent on the test date by marking the first; 
entry in the "For Test Administrator Use Only" area on the answer sheet. 
Make-up administrations were not to be scheduled. (4,414 pupils at 102 
schools actually took the test; there were 306 absentees and 10 invalid 
cases , providing usable data for 4,098 pupils for .further analyses .) 

f t ^ : • 

Post-Admi nistration Activities by Schools and lED/ County Coordinators 

Following the actual test administration, the teachers (test admln- 
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Istrators) were asked to supply Individual pupil data on the front of 
each pupil's answer sheet. These data Included information on whether 
' the pupil was officially classified as handicapped, his/her racial 01} 
national origin,, participation in federally funded programs, ease of ' 
providing information on participation in a migrant pre-school program, 
bilinguaUty (and whether, if pupil was bilingual, this constituted a 
learning handicapped) , whether the pupil had been specifically diagnosed ' 
as needing corrective or remedial work in reading, and whether he/she 
was particl^ting in a corrective or remedial reading program. 

School coordinators were reque^te^ to verify that all studenk-rhad 
gridded proparly, and to be sure that the teachers had suppl'ied all 
individual pupil data. After completing a School Coordinator's Report 
Form, all materials were returned to appropriate lED/ County Coordinator. 
lED/County Coordinators were asked to do additional editing prior to the 
'return of materials to ETS 

f 

As ^ condition of\ the contract with the Oregon State Department of 
Education, lED units weVe reimbursed (only upon specific request indicated 
■ on the lED/County Coordinator's Report Form) for expenses generated in 
the administration of the Assessment program. The maximum amount of reim- 
bursement was* not to exceed $200 plus 25 cents per document edited, up to 
a maximum Of $400. Nine lEDs did not request reimbursement; two others 
did not return the Report Form. 

♦ 

Materials Receipt; Accuracy of lED Editing - , 

When all test materials had been returned by the lED/ County Coordinators 
test books were checked and verified against the materials originally 
shipped. A written report of missing materials was transmitted to the Oregon 
State Department of Education for follow-up and resolution. 
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* As part of its work, ETS was asked to analyze the accuracy of lED/ 
Cotmty Coordinator editing,"^d to compare the results for those who had 
been reimbursed with those who had not requested payment. A con5)arison 
of those lED Coordinators requesting Rayment of an honorarium (N « 18) 
with those not requesting payment (N • 9) appears to suggest, at least 
on a proportional basis, that those who did tiot request payment performed 
th§ editing functipn less accurately than those requesting payment. The 
errors that did ejdst seemed to. cluster in a single school, rather than 
within an individual lED. 



Scoring, Processing, and Reporting Schools and Districts 

All answer sheets were optically scanned, transcribed, and edited 
in accordance with editiing rules ar^d procedures approved by the Oregon 
State Department of Ed^:^tign 

During the cours^ ^| m^Vihine processing^ a sample of approximately 
45 quality control caji^|(ev^ry 100th record) was identified.' These 
cases were hand-process^, ^apd the results compared with the computer ^ 
output. All reports we^r^fe Verified as correct prior to release of actual 

r' J 

score reports to particilpating schools and districts. 

Two output reports were designed by ETS, reviewed, revised, and 

i 

Approved by the Oregon State Department of Education. The Student Report 
consisted of an alphabetic roster of pupil names and, for each individual, 
the number of responses marked Right, Wrong, or Don't Know for the twenty- 
five objectives and the four skills areas. In addition, the number of Word 
Identification items marked "Clear" or "Not Clear" was shown. Absent;ees 
were idep^fied by the legend "Absentee", and any pupil whose test contained 
four or fewer responses was identified with the legend '"Not Attempted". 



The School Report displayed the twenty-five objectives, the four 
"grouped objectives" (i.e., skllla areas), and the Word Identification 
score, and provided, for each heading, the number of questions, the mean 
number or Right responses per pupil and standard deviation for that school. 

The Student and School Reports, together with a document entitled 
"Interprfetlve Guide - Pupil and School Report" were mailed to participating 
schools and district superintendents? 

Based on an ETS magnetic tape containing all data transcribed in the 
course of the program, the Oregon State Department of Education contracted 
for the conduct of an independent verification of three aspects of ETS 
processing^. For a sample of 459 answer sheets selected by the Oregon 
State Department of Education, the data were coded and compared with ETS 
data. In addition, RTI, SDE, and ETS codes wiere matched and verified. 
Finally, the scoring of correct responses, by objectives and skills, was • 
checked. No discrepancies were found in the latter two cases. In per- 
formance of the scanning function, 27 discrepancies were found in almost 
80,000 characters compared (a discrepancy rate of 0.0338%); only one of 
the twenty-seven discrepancies was a direct ETS error not attributable 
to the scanning technique. 

♦ * 

Other Analyses 

Following the reporting of scores, a number of 4ata analyses were 
undertaken. These included the item analysis of the test questions both 
for a| random statewide sample of the 4,0^8 students tested, and for sub- 
samples based on racial subgroups within the population. Intercorrelatlons 
were computed and a factor analysis performed. 
<» 

Memorandum from Nlleen Hunt, Research Triangle Institute, 7-1-74, Verification 
of the 1973-74 Oregon Statewide Assessment Data Tape, 3 pages. 
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Weighted frequency distributions of scores for each reading objective 
and for the four "sklljls" areas were provided on each of the following 
grotq)s: 

a) the total State sample 
•b) -by geographic region 

c) by district wealth 

d) by district size 

e) by district size within geographic region 

f) for the Metropolitan district 

g) by two racial subgroups ^ * 
Sunnoary of Results 

The test for the Oregon Statewide Assessment was designed to measure 
pupil performance on twenty-five reading objectives as well as four basic 
skills: wGrd Attack, Vocabulary, Comprehension, and Applicatior.. These 
skills are usually^ considered to be relatively independent components of 
the reading process, and most reading curricula Involve specific Ins true- 
tlon In each skill area. However, their measurement at the eleiientary 
school level typically reflects a high degree of Interdependence among 
the skills. This Interdependence is reflected in the reading test used 
in the' Oregon program. 

The correlations among the four basic skill areaa are shown iln table 
1, as are th6 reliability estimates (KR 20) for each of the four aleas of 



the test.^ 



Insert Table 1 about here 



Pupil performance is summarized in Table 2. Because each skill area 
contains a different number of test items, the average number of corWect 
responses for each skill area la also expressed as a percentage of' the 
number of items. 

Most standardlzed^esta of reading are designed to have an average 
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difficulty level of 35 to 65 percent of the items. This level is selected 
so that the instrument will yield reliable measurement throughout the 
entire range of reading ability. In the case of the Oregon data, the 
• performance of pupils on the Word Attack, Vocabulary, and Comprehension 

N 

sections significantly exceeds this level. ' Thus, only the implication 
Section of the test has an adequate difficulty level to reliably assess 
the reading skills of the better readers. 



Insert Table 2 about here 



Jt is reasonable to infer that, Ilue to the characteristics of the 
test itself, most pupils found the test relatively easy and that the 
"bright" pupils, therefore, were unable to demonstrate the fiull extent 
of their Reading skills. While this lack of discriminatory power at ' 
the ^^^^ ^® ability range, does not seriously affect the valjldity of 
the^^tst results for heterogeneous groups of pupils, this limitation must be 
ta&en into cpnsideration when interpreting group data and the factor 
aalytic results presented in Table 3. * 

\ 

Insert Table 3 about here 



^ 

p The factor analytic structure of the test (the statistical analysis 
of inter-relationships of the objectives) indicates a strong •'Word Attack" 
fadtor (defined by Objectives 1 through 9), a strong "Comprehension" factor 
(defined by objectives 12, 13, and 15 through 21) and a weak "Application" 
Factor (defined primarily by objectives 23. through 25,' which consist of a 
total of only four test items). The complex structure of the Vocabulary^ 
objectives (as. Indicated on Factors 1 and 2) may be attributed'to the fact 
that the Vocabulary items were easy, i.e., most pupils earned perfect 
scores ^on the Vocabulary objectives. In this situation, the correlation 
J of objectives 10 and 11 with all other objectives would be expected to be 
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essentially of the same magnitude. Factor 2 clearly indicates a high \ " 
degree of interdependence between Comprehension and Application Objectives^ 



Conclusion 

This has been an attecq^t to report. the operational highlights of the 
first or pilot year of Oregon's Statewide Assessment program in reading. We 
are currently in the midst of the first full-scale year of reading assessment 
at Grade 4. As a matter of fact, today is the date on which some 9,000 or 
id, 000 Fourth Graders in Oregon are participating in that test administration. 



c 
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TABLE 1. Number and Performance of Pupils,* llellablllt^ 
Estimates, and Correlations 



SKILL MEA 



N 



Mean S.D. 



Word Attack 4098 

Vocabulary 4098 

Compirehenslon 4098 

Application 4098 



31.1 
6.8 
15.2 
14.8 



6.9 
1.8 
5.4 
5.8 



Reliability 

KR-20 Word Att. 



.89 
.78 
.87 
.88 



CORRELATIONS 



.64 
.67 
.70 



Voc . Comp 

.64 .67 

. .68 

.68 

.63 .77 



.70 
.63 
.77 



TABLE 2. Mean Scores Expressed aar Percentages of Number 
of Items in Each Skil^^ea 



"I 

■SKILL AREA 
'^opk Attack 
;^(Kjabulary 
C^j^rehension 
I Application 



No. of Items ■ • Mean Percentages 

40 31.1 78 

8 6.8 - 85 

;? 21 . 15.2 72 

V-]24 14.8 62 

J 

— ^ 



5. 



if 



II,, 
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TABLE 3. Factor Analytic Structure of the Reading Test 

' ... 

READING OBJECTIVE FACTOR 1 . , FACTOR 2 FACTOR 3 

(Word Attack Objectives) ; 

1 . .56 * ' ' 

2 .68 

3 .58 

A .63 • ^ 

5 . .57 . .32 

6 . .68 

• I 

8 .54 • , 

9 .62 _ • 
(Vocabulary Objectives) * ■ 

10 Ul .54 • ■ . 

■ . • -.34 .57 
(Comprehension Objectives > 

. 'l^' ' , .67 

13 4 .79 

^ ; • - ,36 

H ; ■ .33 

16 

(Application Objectives) ♦ 

18 * .30 .58 

19 .38 .52 - 

20 .31 .71 

21 .49 

22 ^ . .34 

23 7^ 75 

24 , 75 

25 ■ is" 



.75 

,66 





